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An examiner's amendment 

An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 
1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview with 
William Hunter on 4/22/2009. 
Amendment to the specification is as follows: 

Please replace paragraph [0005], beginning at page 2, line 15, with the following amended 
paragraph: 

[0005] The present disclosure includes systems and techniques relating to augmented virtual 
environments , which can be implemented using a machine-readable storage device 
embodying information indicative of instructions for causing one or more machines to 
perform operations as described . According to an aspect, a three dimensional model of an 
environment can be generated from range sensor information representing a height field for the 
environment. Position and orientation information of at least one image sensor in the 
environment can be tracked with respect to the three dimensional model in real-time. Real-time 
video imagery information from the at least one image sensor can be projected onto the three 
dimensional model based on the tracked position and orientation information, and the three 
dimensional model can be visualized with the projected real-time video imagery. Additionally, 
generating the three dimensional model can involve parametric fitting of geometric primitives to 
the range sensor information. 



The amendment filed after final on 9/15/2008 has been entered, the amended claims are 45-48, 
and 50, see below: 

Examiner's notes: the claims amendment filed on 3/13/2008 page 5 disclosed claims 40-49 
cancelled, but Applicant added new claims 41-50, which the added new claims should have been 
numbered as 50-59. The renumbering of claims is corrected in this action. 
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The limitations of dependent claims 41, 43, and 49 are inserted into the body of their 
independent claims 29, 37, and 45, see below: 

25. The method of claim 29, wherein the surface comprises a two dimensional surface. 
29. A method comprising: obtaining a three dimensional model of a three dimensional 
environment, the three dimensional model generated from range sensor information representing 
a height field for the three dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- time video 
imagery information from at least one image sensor having associated position and orientation 
information with respect to the three dimensional model, the background image comprising a 
single distribution background dynamically modeled from a time average of the real-time video 
imagery information; 

placing a surface that corresponds to the moving region in the three dimensional model, wherein 
placing the surface comprises casting a ray from an optical center, corresponding to the real-time 
video imagery information, to a bottom point of the moving region in an image plane in the three 
dimensional model, and determining a position, an orientation and a size of the surface based on 
the ray, a ground plane in the three dimensional model, and the moving region; 
projecting the real-time video imagery information onto the three dimensional model, including 
the surface, based on the position and orientation information; and 

visualizing the three dimensional model with the projected real-time video imagery; wherein 
identifying a region in motion in real time comprises subtracting the background image from the 
real-time video imagery information, identifying a foreground object in the subtracted real-time 
video imagery information, validating the foreground object by correlation matching between 
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identified objects in neighboring image frames, and outputting the validated foreground object^ 
wherein identifying a foreground object comprises identifying the foreground object in the 
subtracted real-time video imagery information using a histogram-based threshold and a 
noise filter . 

30. The method of claim 29, further comprising tracking the position and orientation information 
of the at least one image sensor in the environment with respect to the three dimensional model 
in real-time. 

31. The method of claim 30, wherein obtaining a three dimensional model of a three dimensional 
environment comprises generating the three dimensional model of the three dimensional 
environment. 

33. The system of claim 37, wherein the surface comprises a two dimensional surface. 
37. An augmented virtual environment system comprising: an object detection and tracking 
component that identifies in real time a region in motion with respect to a background image in 
real-time video imagery information from at least one image sensor having associated position 
and orientation information with respect to a three dimensional model of a three dimensional 
environment, the three dimensional model generated from range sensor information representing 
a height field for the three dimensional environment, the background image comprising a single 
distribution background dynamically modeled from a time average of the real-time video 
imagery information, and places a surface that corresponds to the moving region with respect to 
the three dimensional model, wherein the object detection and tracking component places the 
surface by performing operations comprising casting a ray from an optical center, corresponding 
to the real-time video imagery information, to a bottom point of the moving region in an image 
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plane in the three dimensional model, and determining a position, an orientation and a size of the 
surface based on the ray, a ground plane in the three dimensional model, and the moving region; 
a dynamic fusion imagery projection component that projects the real-time video imagery 
information onto the three dimensional model, including the surface, based on the position and 
orientation information; and a visualization sub-system that visualizes the three dimensional 
model with the projected real-time video imagery; wherein the object detection and tracking 
component identifies the moving region by performing operations comprising subtracting the 
background image from the real-time video imagery information, identifying a foreground object 
in the subtracted real-time video imagery information, validating the foreground object by 
correlation matching between identified objects in neighboring image frames, and outputting the 
validated foreground objec t; wherein identifying a foreground object comprises identifying 
the foreground object in the subtracted real-time video imagery information using a 
histogram-based threshold and a noise filter . 

38. The system of claim 37, further comprising a tracking sensor system that integrates visual 
input, global navigational satellite system receiver input, and inertial orientation sensor input to 
obtain the position and the orientation information associated with the at least one image sensor 
in real time in conjunction with the real-time video imagery. 

39. The system of claim 38, further comprising a model construction component that generates 
the three dimensional model of the three dimensional environment. 

41. (Canceled) 

42. The method of claim 41, wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the background image as a temporal 
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pixel average of five recent image frames in the real-time video imagery information. 

43. (Canceled) 

44. The system of claim 43, wherein identifying a region in motion in real time further comprises 
estimating the background image by modeling the background image as a temporal pixel average 
of five recent image frames in the real-time video imagery information. 

45. A machine-readable storage device [medium] embodying information indicative of 
instructions for causing one or more machines to perform operations comprising: 

obtaining a three dimensional model of a three dimensional environment, the three dimensional 
model generated from range sensor information representing a height field for the three 
dimensional environment; 

identifying in real time a region in motion with respect to a background image in real- time video 
imagery information from at least one image sensor having associated position and orientation 
information with respect to the three dimensional model, the background image comprising a 
single distribution background dynamically modeled from a time average of the real-time video 
imagery information; placing a surface that corresponds to the moving region in the three 
dimensional model, wherein placing the surface comprises casting a ray from an optical center, 
corresponding to the real-time video imagery information, to a bottom point of the moving 
region in an image plane in the three dimensional model, and determining a position, an 
orientation and a size of the surface based on the ray, a ground plane in the three dimensional 
model, and the moving region; projecting the real-time video imagery information onto the three 
dimensional model, including the surface, based on the position and orientation information; and 
visualizing the three dimensional model with the projected real-time video imagery; wherein 
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identifying a region in motion in real time comprises subtracting the background image from the 
real-time video imagery information, identifying a foreground object in the subtracted real-time 
video imagery information, validating the foreground object by correlation matching between 
identified objects in neighboring image frames, and outputting the validated foreground object^ 
wherein identifying a foreground object comprises identifying the foreground object in the 
subtracted real-time video imagery information using a histogram-based threshold and a 
noise filter . 

46. The machine -readable storage device [medium] of claim 45, wherein the surface comprises 
a two dimensional surface. 

47. The machine-readable storage device [medium] of claim 45, further comprising tracking the 
position and orientation information of the at least one image sensor in the environment with 
respect to the three dimensional model in real-time. 

48. The machine-readable storage device [medium] of claim 47, wherein obtaining a three 
dimensional model of a three dimensional environment comprises generating the three 
dimensional model of the three dimensional environment. 

49. (Cancelled) 

50. The machine -readable storage device [medium] of claim 45, wherein identifying a region in 
motion in real time further comprises estimating the background image by modeling the 
background image as a temporal pixel average of five recent image frames in the real-time video 
imagery information. 



Application/Control Number: 10/676,377 Page 8 

Art Unit: 2628 

Allowable Subject Matter 

Claims 25, 29-31, 33, 37-39, 42, 44-48, and 50 are allowed. 

The following is an examiner's statement of reasons for allowance: 

The cited prior arts do not teach or suggest "placing a surface that corresponds to the 
moving region in the three dimensional model, wherein placing the surface comprises casting a 
ray from an optical center, corresponding to the real-time video imagery information, to a bottom 
point of the moving region in an image plane in the three dimensional model, and determining a 
position, an orientation and a size of the surface based on the ray, a ground plane in the three 
dimensional model, and the moving region; projecting the real-time video imagery information 
onto the three dimensional model, including the surface, based on the position and orientation 
information; and visualizing the three dimensional model with the projected real-time video 
imagery; wherein identifying a region in motion in real time comprises subtracting the 
background image from the real-time video imagery information, identifying a foreground object 
in the subtracted real-time video imagery information, validating the foreground object by 
correlation matching between identified objects in neighboring image frames, and outputting the 
validated foreground object; wherein identifying a foreground object comprises identifying the 
foreground object in the subtracted real-time video imagery information using a histogram-based 
threshold and a noise filter", which is recited in claim 29, above. 

The cited prior arts do not teach or suggest "places a surface that corresponds to the 
moving region with respect to the three dimensional model, wherein the object detection and 
tracking component places the surface by performing operations comprising casting a ray from 
an optical center, corresponding to the real-time video imagery information, to a bottom point of 
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the moving region in an image plane in the three dimensional model, and determining a position, 
an orientation and a size of the surface based on the ray, a ground plane in the three dimensional 
model, and the moving region; a dynamic fusion imagery projection component that projects the 
real-time video imagery information onto the three dimensional model, including the surface, 
based on the position and orientation information; and a visualization sub-system that visualizes 
the three dimensional model with the projected real-time video imagery; wherein the object 
detection and tracking component identifies the moving region by performing operations 
comprising subtracting the background image from the real-time video imagery information, 
identifying a foreground object in the subtracted real-time video imagery information, validating 
the foreground object by correlation matching between identified objects in neighboring image 
frames, and outputting the validated foreground object; wherein identifying a foreground object 
comprises identifying the foreground object in the subtracted real-time video imagery 
information using a histogram-based threshold and a noise filter", which is recited in claim 37, 
above. 

The cited prior arts do not teach or suggest "placing a surface that corresponds to the 
moving region in the three dimensional model, wherein placing the surface comprises casting a 
ray from an optical center, corresponding to the real-time video imagery information, to a bottom 
point of the moving region in an image plane in the three dimensional model, and determining a 
position, an orientation and a size of the surface based on the ray, a ground plane in the three 
dimensional model, and the moving region; projecting the real-time video imagery information 
onto the three dimensional model, including the surface, based on the position and orientation 
information; and visualizing the three dimensional model with the projected real-time video 
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imagery; wherein identifying a region in motion in real time comprises subtracting the 
background image from the real-time video imagery information, identifying a foreground object 
in the subtracted real-time video imagery information, validating the foreground object by 
correlation matching between identified objects in neighboring image frames, and outputting the 
validated foreground object; wherein identifying a foreground object comprises identifying the 
foreground object in the subtracted real-time video imagery information using a histogram-based 
threshold and a noise filter", which is recited in claim 45 above. 

The dependent claims 25, 30-31, 33, 38-39, 42, 44, 46-48, and 50 are allowed with the 
same reasons as set forth in their independent claims 29, 37 and 45, above. 

The closest prior art is Sawhney et al. (hereinafter Sawhney) that presents an immersive 
model-based video visualization system, called the Video Flashlight system, that provides an 
augmented reality solution for video surveillance and monitoring applications. The system 
enables active browsing and visualization of a 3D model of a large scale site by rendering 
multiple videos from a blanket of ground and aerial video cameras over the model. By 
seamlessly rendering dynamic video data from multiple cameras on top of a 3D model of a site, 
the system allows the users to view the dynamic action in the context of a global 3D model while 
actively viewing the integrated model and videos from the viewpoint of a virtual camera. Thus, 
the system supports seamless viewpoint change for a sky-to-street level browsing mode with 
integrated views of the model and the videos. However, Sawhney does not teach or suggest the 
3D model including a surface based on the position and orientation information, and identifying 
a foreground object comprises identifying the foreground object in the subtracted real-time video 
imagery information using a histogram-based threshold and a noise filter. 
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Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JAVID A. AMINI whose telephone number is (571)272-7654. 
The examiner can normally be reached on 8-4pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kee Tung can be reached on 571-272-7794. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Javid A Amini 
Primary Examiner 
Art Unit 2628 
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Primary Examiner, Art Unit 2628 



